Gong, Zhenhuan. Multi-level Data Layout Optimization for Heterogeneous Access Patterns. (under the Direction of Dr. Nagiza F. Samatova.) Multi-level Data Layout Optimization for Heterogeneous Access Patterns

GONG, ZHENHUAN. Multi-level Data Layout Optimization for Heterogeneous Access Patterns. (Under the direction of Dr. Nagiza F. Samatova.) Recent years have seen an enormous increase in computation power of leadership computing facilities. As a result, huge amounts of data, from terascale to petascale, are being produced by scientific applications running on supercomputers. However, the I/O subsystems have not developed with a comparable speed, making data I/O and storage the major bottleneck in modern computing architectures. The problem gets exacerbated by the need to perform data-intensive analytic jobs, such as queries with multiple constraints on these datasets stored on external storage. Scientific applications produce multi-dimensional, multi-variate, double-precision datasets, and these datasets are usually stored on large-scale parallel file systems. The datasets are not well-represented by traditional relational data models. Queries on scientific datasets involve multiple constraints, thus producing heterogeneous I/O access patterns. How extreme-scale datasets are linearized and organized on parallel file systems is crucial to the data read performance for queries: the optimized layout results in more sequential reads on contiguous data blocks, which are much faster than seek-and-reads on non-contiguous small blocks. Existing data layout optimization techniques, while successfully improved read performance for certain application-specific access patterns, have failed to address more general and heterogeneous access patterns. They also often do not scale to the expected substantial growth in data size reaching exascale in the near future. Moreover, existing technology usually performs data layout optimization in a post-processing way on datasets on storage systems. The post-processing approaches read the entire datasets on storage, perform layout optimization, and write the processed data back to storage, which is extremely inefficient due to the I/O bottleneck in modern computer architecture, and the huge size of the datasets. There is a lack of a general framework to perform data layout optimization for scientific datasets at simulation run time or I/O time, before datasets are written to storage, to reduce I/O and storage overhead, and speed up the entire process. To address the problems, a multi-level data storage layout scheme is presented, which optimizes for heterogeneous access patterns induced by different types of queries for scientific data analysis. First, a hybrid layout scheme is presented, which is optimized for two common access patterns: value-constrained accesses and space-constrained accesses. The layout scheme improves data locality and reduces the latency bound I/O operations (such as seeks) substantially. This layout scheme is further generalized by MLOC, a parallel M ultilevel Layout Optimization framework for C ompressed scientific spatio-temporal data at extreme scale. MLOC includes multiple fine-grained data layout optimization kernels that form a generic core from which a broader constellation of such kernels can be organically consolidated in a hierarchical multilevel architecture, in order to enable an effective data exploration with various combinations of access patterns. Specifically, the kernels are optimized for access patterns induced by (a) query-driven multi-variate, spatio-temporal constraints, (b) precision-driven data analytics, (c) compression-driven data reduction, (d) multi-resolution data sampling, and (e) multi-file data partitioning and organization on parallel file systems. When tested on query-driven exploration of compressed data, MLOC demonstrates an order of magnitude faster query response time compared to the state-of-the-art scientific database management technology. Based on MLOC, a parallel run-time data layout optimization framework, PARLO, is presented to perform data layout optimization on scientific datasets at simulation run time, By processing the datasets when they are still in memory before written to disks, PARLO successfully removes additional I/O and storage overhead, and significantly reduces overall processing time compared with traditional post-processing approaches. PARLO is integrated with ADIOS, a popular parallel I/O middleware. With support of ADIOS’s I/O libraries and the portable BP file formats, PARLO achieves high-performance, parallel, and user-transparent data layout optimization at run time. c © Copyright 2013 by Zhenhuan Gong